Implicit Kernel Attention

Attention computes the dependency between representations, and it encourages model to focus on important selective features. Attention-based models, such as Transformer graph attention network (GAT), are widely utilized for sequential data graph-structured data. This paper suggests a new interpretation generalized structure of in GAT. For GAT, we derive that is product two parts: 1) RBF kernel measure similarity instances 2) exponential L2 norm compute importance individual instances. From this decomposition, generalize three ways. First, propose implicit with an function instead manual selection. Second, Lp norm. Third, extend our structured multi-head attention. Our shows better performance classification, translation, regression tasks.